7 research outputs found
Effective and Efficient Query-aware Snippet Extraction for Web Search
Query-aware webpage snippet extraction is widely used in search engines to
help users better understand the content of the returned webpages before
clicking. Although important, it is very rarely studied. In this paper, we
propose an effective query-aware webpage snippet extraction method named
DeepQSE, aiming to select a few sentences which can best summarize the webpage
content in the context of input query. DeepQSE first learns query-aware
sentence representations for each sentence to capture the fine-grained
relevance between query and sentence, and then learns document-aware
query-sentence relevance representations for snippet extraction. Since the
query and each sentence are jointly modeled in DeepQSE, its online inference
may be slow. Thus, we further propose an efficient version of DeepQSE, named
Efficient-DeepQSE, which can significantly improve the inference speed of
DeepQSE without affecting its performance. The core idea of Efficient-DeepQSE
is to decompose the query-aware snippet extraction task into two stages, i.e.,
a coarse-grained candidate sentence selection stage where sentence
representations can be cached, and a fine-grained relevance modeling stage.
Experiments on two real-world datasets validate the effectiveness and
efficiency of our methods.Comment: Accepted by EMNLP202
LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval
In large-scale retrieval, the lexicon-weighting paradigm, learning weighted
sparse representations in vocabulary space, has shown promising results with
high quality and low latency. Despite it deeply exploiting the
lexicon-representing capability of pre-trained language models, a crucial gap
remains between language modeling and lexicon-weighting retrieval -- the former
preferring certain or low-entropy words whereas the latter favoring pivot or
high-entropy words -- becoming the main barrier to lexicon-weighting
performance for large-scale retrieval. To bridge this gap, we propose a
brand-new pre-training framework, lexicon-bottlenecked masked autoencoder
(LexMAE), to learn importance-aware lexicon representations. Essentially, we
present a lexicon-bottlenecked module between a normal language modeling
encoder and a weakened decoder, where a continuous bag-of-words bottleneck is
constructed to learn a lexicon-importance distribution in an unsupervised
fashion. The pre-trained LexMAE is readily transferred to the lexicon-weighting
retrieval via fine-tuning. On the ad-hoc retrieval benchmark, MS-Marco, it
achieves 42.6% MRR@10 with 45.8 QPS for the passage dataset and 44.4% MRR@100
with 134.8 QPS for the document dataset, by a CPU machine. And LexMAE shows
state-of-the-art zero-shot transfer capability on BEIR benchmark with 12
datasets.Comment: Appeared at ICLR 202
Inference with Reference: Lossless Acceleration of Large Language Models
We propose LLMA, an LLM accelerator to losslessly speed up Large Language
Model (LLM) inference with references. LLMA is motivated by the observation
that there are abundant identical text spans between the decoding result by an
LLM and the reference that is available in many real world scenarios (e.g.,
retrieved documents). LLMA first selects a text span from the reference and
copies its tokens to the decoder and then efficiently checks the tokens'
appropriateness as the decoding result in parallel within one decoding step.
The improved computational parallelism allows LLMA to achieve over 2x speed-up
for LLMs with identical generation results as greedy decoding in many practical
generation scenarios where significant overlap between in-context reference and
outputs exists (e.g., search engines and multi-turn conversations).Comment: 9 page
VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
Although speech is a simple and effective way for humans to communicate with
the outside world, a more realistic speech interaction contains multimodal
information, e.g., vision, text. How to design a unified framework to integrate
different modal information and leverage different resources (e.g.,
visual-audio pairs, audio-text pairs, unlabeled speech, and unlabeled text) to
facilitate speech representation learning was not well explored. In this paper,
we propose a unified cross-modal representation learning framework VATLM
(Visual-Audio-Text Language Model). The proposed VATLM employs a unified
backbone network to model the modality-independent information and utilizes
three simple modality-dependent modules to preprocess visual, speech, and text
inputs. In order to integrate these three modalities into one shared semantic
space, VATLM is optimized with a masked prediction task of unified tokens,
given by our proposed unified tokenizer. We evaluate the pre-trained VATLM on
audio-visual related downstream tasks, including audio-visual speech
recognition (AVSR), visual speech recognition (VSR) tasks. Results show that
the proposed VATLM outperforms previous the state-of-the-art models, such as
audio-visual pre-trained AV-HuBERT model, and analysis also demonstrates that
VATLM is capable of aligning different modalities into the same space. To
facilitate future research, we release the code and pre-trained models at
https://aka.ms/vatlm.Comment: 10 page
Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark
Large language models (LLMs) have demonstrated powerful capabilities in both
text understanding and generation. Companies have begun to offer Embedding as a
Service (EaaS) based on these LLMs, which can benefit various natural language
processing (NLP) tasks for customers. However, previous studies have shown that
EaaS is vulnerable to model extraction attacks, which can cause significant
losses for the owners of LLMs, as training these models is extremely expensive.
To protect the copyright of LLMs for EaaS, we propose an Embedding Watermark
method called EmbMarker that implants backdoors on embeddings. Our method
selects a group of moderate-frequency words from a general text corpus to form
a trigger set, then selects a target embedding as the watermark, and inserts it
into the embeddings of texts containing trigger words as the backdoor. The
weight of insertion is proportional to the number of trigger words included in
the text. This allows the watermark backdoor to be effectively transferred to
EaaS-stealer's model for copyright verification while minimizing the adverse
impact on the original embeddings' utility. Our extensive experiments on
various datasets show that our method can effectively protect the copyright of
EaaS models without compromising service quality.Comment: Accepted by ACL 202